Apparatus and method for normalizing face image used for detecting drowsy driving

ABSTRACT

Disclosed are an apparatus and a method for normalizing an image of a driver&#39;s face in a predetermined size on the center of a monitor by automatically controlling a lens when the image of the driver&#39;s face is detected. The apparatus includes a lens for photographing a driver&#39;s face, a first motor for moving the lens in a forward or rearward direction in order to adjust a zoom parameter, a second motor for moving the lens in a horizontal or vertical direction in order to adjust a pan parameter or a tilt parameter, and a controller for extracting an initial face area from a photographed image and controlling operations of the first motor and the second motor according to the extracted initial face area.

PRIORITY

This application claims priority to an application entitled “Apparatusand Method for Normalizing Face Image Used for Detecting Drowsy Driving”filed in the Korean Intellectual Property Office on Jan. 20, 2005 andassigned Serial No. 2005-5555, the contents of which are incorporatedherein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus and a method forextracting a face image required for detecting drowsy driving, and moreparticularly to an apparatus and a method for controlling a lens so thatan image of a driver's face can be normalized to a predetermined size inthe center of a monitor.

2. Description of the Related Art

In order to detect drowsy driving, a procedure of extracting images ofdrivers' faces must be first performed. How this procedure is performedcan substantially affect the overall performance of a conventionalapparatus for detecting drowsy driving. However, since the drivers mayassume various driving positions and postures (e.g., the drivers may situpright, reclined, set-back, set-forward, slouched, etc.), it isdifficult to accurately detect the images of the drivers' faces. Inaddition, since the drivers' faces have different sizes, detecting theimages of the driver's face becomes more difficult.

Images of a driver's feature photographed by the conventional apparatusfor detecting drowsy driving having a fixed camera are shown in FIGS. 1(a)-1(d).

FIG. 1( a) illustrates a normal image, and FIGS. 1( b) to 1(d)illustrates abnormal images; FIG. 1( b) illustrates a case where animage is abnormal because the driver's face is photographed as a verysmall image; FIG. 1( c) illustrates a case where an image is abnormalbecause the driver's face is photographed as a very large image; andFIG. 1( d) illustrates a case where the image is abnormal because thedriver's face is photographed as an image excessively-offset to theleft.

When the apparatus for detecting drowsy driving is initially installed,the position for mounting the camera must be determined according toindividual drivers (and their driving habits) in such a manner that theabnormal images describe above are not obtained. Although the camera issuitably mounted, driving habits may change somewhat, and/or otherdrivers may drive the vehicle. However, it is difficult to control theposition of the camera or perform re-calibration whenever a driver'sdriving habit changes somewhat, and/or other drivers drive the vehicle.

The problems are caused because the conventional apparatus for detectingdrowsy driving extracts the image of the driver's face by employing afixed camera.

In addition, the apparatus for detecting drowsy driving employing afixed camera recognizes the distance between the camera and the driveras an average distance, and an operation of extracting an areacorresponding to the image of the driver's face based on thephotographed image is performed through software, which increases thetime required to extract an image and reduces the system's efficiency.

In addition, if the image of the driver's face is intended to beextracted from an image using the fixed camera, a normalizationprocedure must be performed. Although the normalization procedure isperformed, information about the resolution of an input face image isalways changing. Herein, the normalization procedure is required becausethe face area in the photographed image may have a different position ora different size due to a difference in a face's position with respectto a camera lens according to the driver's driving position.

SUMMARY OF THE INVENTION

Accordingly, the present invention has been made to solve theabove-mentioned problems occurring in the prior art, and an object ofthe present invention is to provide an apparatus and a method, whichallow a driver's face to be positioned at the center of a photographedimage in a predetermined size through the adjustment of a pan parameter,a tilt parameter, or a zoom parameter, thereby enhancing the performanceand the speed for extracting the image of the driver's face anddetecting drowsy driving.

To accomplish the above objects, there is provided an apparatus forextracting and normalizing a face image to detect drowsy driving, theapparatus including a lens for photographing a driver's face, a firstmotor for moving the lens in a forward or rearward direction in order toadjust a zoom parameter, a second motor for moving the lens in ahorizontal or vertical direction in order to adjust a pan parameterand/or a tilt parameter, and a controller for extracting an initial facearea from a photographed image and controlling operations of the firstmotor and the second motor according to the extracted initial face area.

According to another aspect of the present invention, there is provideda method for normalizing face image extraction in an apparatus forextracting the face image used for detecting drowsy driving, whichincludes a lens enabling pan, tilt, and/or zoom parameter adjustment,the method including the steps of extracting an initial face area froman input image, calculating face elements based on the extracted initialface area, determining if adjustment of a pan parameter, a tiltparameter, and/or a zoom parameter is required by analyzing results ofthe calculated face elements, and obtaining a normalized face image byadjusting the pan parameter, the tilt parameter, and/or the zoomparameter according to the determination result.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will be more apparent from the following detailed descriptiontaken in conjunction with the accompanying drawings, in which:

FIGS. 1( a) to 1(d) are images illustrating a driver's featurephotographed by the conventional apparatus for detecting drowsy drivinghaving a fixed camera.

FIG. 2 is a block diagram illustrating an apparatus for extracting andnormalizing a face image used for detecting drowsy driving according toa preferred embodiment of the present invention.

FIGS. 3( a) to 3(d) are images illustrating a method for extracting andnormalizing a face image used for detecting drowsy driving according toa preferred embodiment of the present invention;

FIGS. 4( a) to (g) illustrate examples of images for explaining aprocedure of calculating a face element in a procedure of extracting andnormalizing a face image used for detecting drowsy driving in detailaccording to a preferred embodiment of the present invention;

FIG. 5 is a flowchart illustrating a method for extracting andnormalizing a face image used for detecting drowsy driving according toa preferred embodiment of the present invention; and

FIG. 6 is a flowchart illustrating a procedure of calculating a faceelement in detail.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings.Although many specific items, such as detailed pixel numbers, detailedinput images, etc., are shown in the following description, these areprovided for the purpose of overall comprehension about the presentinvention. Therefore, it is generally known to those skilled in the artthat the present invention can be embodied without being limited bythese specific items. In the following description of the presentinvention, a detailed description of known functions and configurationsincorporated herein will be omitted when it may make the subject matterof the present invention unclear.

FIG. 2 is a block diagram illustrating an apparatus for extracting andnormalizing a face image used for detecting drowsy driving according toa preferred embodiment of the present invention.

A lens module 220 is used for photographing a subject (a driver's face).A first motor 240 is used for moving the lens module 220 in a forward orrearward direction in order to adjust a zoom parameter. A second motor260 is used for moving the lens module 220 in a horizontal direction orin a vertical direction in order to adjust a pan parameter or a tiltparameter. A memory module 280 is used for storing a program fordetecting drowsy driving, frontal face templates, and a look-up table. Acontroller 200 is used for controlling the driving of the first motorand the second motor 240 and 260, respectively, depending on the look-uptable. A reference numeral Z represents zoom parameter adjustmentcontrol, and a reference numeral P/C represents pan or tilt parameteradjustment control.

FIGS. 3( a) to 3(d) are sample images for illustrating a method forextracting and normalizing a face image used for detecting drowsydriving according to the present invention.

FIG. 3( a) illustrates an input image, and the image is an abnormalimage because a driver's face in the image is small and is offset to theleft lower part of the image.

FIG. 3( b) illustrates an image obtained by performing pan parameteradjustment with respect to the input image shown in FIG. 3( a), and theobtained image is an abnormal image because the driver's face in theimage is undesirably small and is offset to the left lower part of theimage.

FIG. 3( c) illustrates an image obtained by performing tilt parameteradjustment with respect to the image shown in FIG. 3( b). Note thisimage is still an abnormal image because the driver's face in the imageis still small.

FIG. 3( d) illustrates an image obtained by performing zoom parameteradjustment with respect to the image shown in FIG. 3( c), and the imageis now a normal image.

FIGS. 4( a) to 5(g) illustrate examples of images for describing aprocedure of calculating a face element in a procedure of extracting andnormalizing a face image used for detecting drowsy driving in detailaccording to the present invention.

FIG. 4( a) illustrates an input image; FIG. 4( b) illustrates an imageobtained by extracting a horizontal edge from the input image shown inFIG. 4( a); FIG. 4( c) illustrates an image obtained by extracting eyeareas and a mouth area from the image shown in FIG. 4( b); FIG. 4( d)illustrates an image obtained by extracting main horizontal lines andmain vertical lines from the image shown in FIG. 4( c); FIG. 4( e)illustrates an image obtained by extracting a main face area from theimage shown in FIG. 4( d); FIG. 4( f) illustrates main coordinates ofthe eye areas and the mouth area required for extracting the mainhorizontal lines and the main vertical lines of FIG. 4( d) and FIG. 4(g) illustrates two coordinates required for extracting a main area.

FIG. 5 is a flowchart illustrating a method for extracting andnormalizing a face image used for detecting drowsy driving according tothe present invention.

In order to stably extract an image area of a driver's face in a vehicleand then extract an eye area based on an extracted face area, the facearea must be detected on an exact position and in a suitable size. Ifthe face area is extracted as a very small area, the extractioninformation about the eye area is inaccurate. If the face area isextracted as very large area, the sight of the eye area is lost when thedriver's face moves, so that the extraction of the eye area can fail. Inaddition, if the face is excessively offset to the left or the right inan image, it is difficult to stably extract the face area. However, ifpan, tilt, or zoom parameter adjustment is performed by automaticallymoving the lens as described later, the image of the driver's face maybe stably extracted.

In step 510, an image is input into the apparatus in an initial state.The initial state denotes a state, in which the driver's face isphotographed in a vehicle by photographing a subject in a large size(e.g., close-up) using a zoom parameter of the lens.

In step 520, the apparatus for extracting and normalizing a face imageused for detecting drowsy driving extracts an initial face area from thereceived image. In order to conveniently and quickly perform acalculation of the initial face area, a face candidate area is searchedaccording to whether the image matches with a low resolution frontalface templates.

In step 530, it is determined if the extraction of the initial face areais successively achieved. In other words, it is determined if the facecandidate area is a face image. For example, the face candidate area ismatched to an original resolution frontal face template. Alternatively,if the initial face area is not found, a procedure for extracting andnormalizing a face image returns to the initial state in step 510. Incontrast, if the initial face area is successively found, the apparatusfor extracting and normalizing a face image returns to step 540 so as toperform a face element calculation of calculating feature elements forthe initial face area.

FIG. 6 is a flowchart illustrating the face element calculation of FIG.5 in detail.

In step 42, a horizontal edge is extracted in order to detect an eyearea and a mouth area from the face area (the initial area having beenpreviously extracted in step 520 of FIG. 5). For example, the extractedhorizontal edge may be the horizontal edge marked in the image shown inFIG. 4( b). Such a horizontal edge may be extracted using a Sobel edgecalculation scheme, which is well known in the art, or a templatematching scheme.

In step 43, in order to extract the eye area, a horizontal edge image isextracted. Horizontal edges of several lines shown under edges of aneyebrow area are recognized as the eye area. In this case, the extractedarea is constructed in the optimal square shape. The eye area may beextracted through a histogram analysis or the template matching scheme.

In step 44, in order to extract the mouth area, a horizontal edge imageis extracted. The mouth area may be extracted based on the horizontaledges shown under edges of a nose area in the center of the face image.In this case, the extracted area is constructed in the optimal squareshape. The mouth area may be detected through a histogram analysis orthe template matching scheme.

In step 45, the main vertical lines are calculated. The main verticallines are calculated on the assumption that the received image has2-dimensional coordinates, and the most left of the upper part of thereceived image is a reference point (0,0). Referring to FIG. 4( d), themain vertical lines correspond to a left vertical line (vertical_left)and a right vertical line (vertical_right). Referring to FIG. 4( f), theleft vertical line (vertical_left) may be defined using a coordinatevalue for a right point (e.g., see P1 or P2 of FIG. 4( f)) of a left eyearea, and the right vertical line (vertical-right) may be defined usinga coordinate value for a left point (e.g., see P4 or P5 of FIG. 4( f))of a right eye area.

In step 46, the main horizontal lines are calculated. The mainhorizontal lines are calculated using coordinates of the eye areas andthe mouth area. The main horizontal lines correspond to an upperhorizontal line (horizontal_up) and a lower horizontal line(horizontal_down). Referring to FIG. 4( f), the upper horizontal line(horizontal_up) may be defined using coordinate values for a lower point(P2, P3, P5, or P6) of the eye areas, and the lower horizontal line(horizontal_down) may be defined using coordinate values for an upperpoint (P7 or P8) of the mouth area.

In step 47, the main area is calculated. The main area is obtained inthe form of a minimum outline square including the eye areas and themouth area. The main area is shown in FIG. 4( e). As shown in FIG. 4(g), the main area may be expressed as two points. The point P9 and thepoint P10 express a square as the left-upper point and the right-lowerpoint, respectively. A coordinate value P9.left represents the smallervalue among an x coordinate value of a left eye area and an x coordinatevalue of a left part of the mouth area. A coordinate value P9.toprepresents the smaller value among a y coordinate value of a left eyearea and a y coordinate value of a right eye area. A coordinate valueP10.right represents the larger value among an x coordinate value of aright eye area and an x coordinate value of a right part of the moutharea. A coordinate value P10.bottom represents a y coordinate value ofthe mouth area.

The main horizontal/vertical lines formed as described above may be usedfor adjusting a pan/tilt parameter, and the main area information may beused for calculating a zoom parameter.

Referring back to FIG. 5, in step 550, the pan parameter is adjusted.For example, the pan parameter adjustment is used for converting theimage shown in FIG. 3(a) into the image shown in FIG. 3( b).

Since the pan parameter information is information varying in ahorizontal direction in an image, main vertical components are used. Inother words, in a QVGA (Quarter Video Graphic Array) image having320×240 pixels, a horizontal component corresponds to 320, and thecenter value corresponds to 160. Since main vertical lines define theinterval between two eyes, the pan parameter value is adjusted in such amanner that the center value of the interval between the two eyesbecomes the pixel value of 160.

On the assumption that the center value of x coordinates values is“meanx”, the pan parameter is adjusted in such a manner that thefollowing equation is satisfied.vertical_left<meanx<vertical_right

In order to adjust parameters, operation angles of motors for adjustingthe lens are adjusted according to the movement of a predetermined pixelin an image. Motor values may be previously defined through the look-uptable.

In step 560, the tilt parameter is adjusted. For example, the tiltparameter adjustment is used for converting the image shown in FIG. 3(b) into the image shown in FIG. 3( c).

In other words, in a QVGA image having 320×240 pixels, a verticalcomponent corresponds to 240, and the center value of the verticalcomponent corresponds to 120. Since main horizontal lines are formed byan upper line and a lower line, the tilt parameter value is adjusted insuch a manner that the center value of the upper line and the lower linebecomes the pixel value of 120.

On the assumption that the center value of y coordinate values is“meany”, the tilt parameter is adjusted in such a manner that thefollowing equation is satisfied.horizontal_top<meany<horizontal_down

The look-up table is used as in the pan parameter adjusting scheme. Ifthe adjustment of the tilt parameter is finished, the zoom parameter isadjusted in step 570. For example, the zoom parameter adjustment is usedfor converting the image shown in FIG. 3( c) into the image shown inFIG. 3( d).

The face image is previously positioned at the center of the receivedimage through the pan/tilt parameter adjustment, and then, the eyes ofthe face image may be stably extracted. However, the zoom parameteradjustment is required such that a driver's face can be stably extractedeven when the driver is drowsy or moves somewhat. The size of the mainarea in the received image is calculated in such a manner that the sizeof the main area is within the range of reference values. Herein, thereference values may be determined by a user. On the assumption that thesize of the main area corresponds to n×m=nm pixels, the zoom parameteris adjusted in such a manner that the following equation is satisfied.Herein, the Min and Max are the reference values used for determiningthe range of the size of the main area.Min<nm<Max

Although it may vary depending on characteristics of cameras, if a faceimage photographed by a camera for detecting drowsy driving has at least64×64 pixels, the size of the main area should be at least 4,096 pixels.However, on the assumption that the allowance corresponds to 100 pixels,the size of the main area may have a value within the range of 3,996pixels to 4,196 pixels. Accordingly, if the size of the extracted mainarea does not correspond to the value within the range of 3,996 pixelsto 4,196 pixels, the zoom parameter is adjusted in such a manner thatthe size of the extracted main area corresponds to the value within therange of 3,996 pixels to 4,196 pixels.

Although all parameters of pan, tilt, and zoom parameters are adjustedas shown in FIGS. 3 to 5 according to an embodiment of the presentinvention, parameters necessary to be adjusted among the pan, tilt, andzoom parameters are first determined by analyzing the result of the faceelement calculation, and then only the determined parameters may beadjusted.

As described above, according to the present invention, a face image isnormalized in such a manner that the face image is extracted in apredetermined size and on a predetermined position regardless of thesize of a driver's face and a driver's driving habit. Accordingly, it ispossible to exactly extract a face image used for detecting drowsydriving.

While the invention has been shown and described with reference tocertain preferred embodiments thereof, it will be understood by thoseskilled in the art that various changes in form and details may be madetherein without departing from the spirit and scope of the invention.Consequently, the scope of the invention should not be limited to theembodiments, but should be defined by the appended claims andequivalents thereof.

1. An apparatus for extracting and normalizing a face image to detect adriver's state of alertness, the apparatus comprising: a lens forphotographing a driver's face; a first motor for moving the lens in aforward or rearward direction in order to adjust a zoom parameter; asecond motor for moving the lens in a horizontal or vertical directionin order to adjust a pan parameter or a tilt parameter; and a controllerfor extracting an initial face area from a photographed driver's faceimage which is an input image and controlling operations of the firstmotor and the second motor if normalization of the face image isrequired by analyzing the extracted initial face area, wherein the panparameter is adjusted by controlling movement of the lens with thesecond motor until a central x coordinate is positioned between two mainvertical lines of the initial face area.
 2. The apparatus as claimed inclaim 1, further comprising a memory storing frontal face templates usedfor extracting the initial face image area through matching in the inputimage.
 3. The apparatus as claimed in claim 1, further comprising amemory for storing a look-up table having a zoom parameter adjustmentvalue used to control the first motor and a pan parameter adjustmentvalue and a tilt parameter adjustment value used to control the secondmotor.
 4. The apparatus as claimed in claim 1, wherein at least one of apan parameter adjustment, a tilt parameter adjustment, and a zoomparameter adjustment is based on a result of face element calculation,the face clement calculation being performed by extracting a horizontaledge from the initial face area extracted from the input image,extracting eye areas and a mouth area, calculating main vertical linesand main horizontal lines, and thereafter calculating a main area. 5.The apparatus as claimed in claim 4, wherein the main vertical lines aredetermined using coordinate values of at least one right most point in aleft eye area of the initial face area and coordinate values of at leastone left most point in a right eye area of the initial face area.
 6. Theapparatus as claimed in claim 4, wherein the main area is determined asa minimum outline square including the eye areas and the mouth area. 7.The apparatus as claimed in claim 6, wherein the main area is defined bymeans of a first point and a second point, an x coordinate value of thefirst point is a smaller value selected from between an x coordinatevalue of a left eye area and an x coordinate value of a left part of themouth area, a y coordinate value of the first point is a smaller valueselected from between a y coordinate value of the left eye area and a ycoordinate value of a right eye area, an x coordinate value of thesecond point is a larger value selected from between an x coordinatevalue of the right eye area and an x coordinate value of a right part ofthe mouth area, and a y coordinate value of the second point is a ycoordinate value of the mouth area.
 8. The apparatus as claimed in claim1, wherein the tilt parameter adjustment is achieved by controllingmovement of the lens with the second motor until a central y coordinateis positioned between two main horizontal lines of the initial facearea.
 9. The apparatus as claimed in claim 4, wherein the mainhorizontal lines are determined using coordinate values of lower pointsin eye areas of the initial face area and coordinate values of upperpoints in a mouth area of the initial face area.
 10. The apparatus asclaimed in claim 1, wherein at least one of the pan parameteradjustment, the tilt parameter adjustment, and the zoom parameteradjustment is based on a result of a face element calculation performedby extracting a horizontal edge from the initial face area extractedfrom the input image, extracting eye areas and a mouth area, and thencalculating a main area, the pan parameter adjustment is performed bycontrolling the second motor such that a center value of x coordinatevalues in the initial face area is positioned between coordinate valuesof at least one right point in a left eye area and coordinate values ofat least one left point in a right eye area, and the tilt parameteradjustment is performed by controlling the second motor such that acenter value of y coordinate values in the initial face area is betweencoordinate values of at least one lower point in the eye areas andcoordinate values of at least one upper point in the mouth area.
 11. Amethod for normalizing face image extraction in an apparatus forextracting the face image used for detecting a driver's state ofalertness, which includes a camera having lens enabling pan, tilt, andzoom parameter adjustment, the method comprising the steps of:extracting an initial face area from an input image, via a controller ofthe apparatus; calculating face elements based on the extracted initialface area, via the controller of the apparatus; determining ifadjustment of a pan parameter, a tilt parameter, and a zoom parameterare required by analyzing results of the calculated face elements, viathe controller of the apparatus; and obtaining a normalized face imageby adjusting at least one of the pan parameter, the tilt parameter, andthe zoom parameter according to the determination result, wherein thepan parameter is adjusted by controlling movement of the lens in apredetermined direction until a central x coordinate is positionedbetween two main vertical lines of the initial face area.
 12. The methodas claimed in claim 11, wherein, in the step of calculating the faceelements includes: extracting a horizontal edge from an initial faceimage extracted from the input image; extracting eye areas and a moutharea from a result of the extracted horizontal edge; calculating mainvertical lines and main horizontal lines based on the extracted eyeareas and the extracted mouth area; and calculating a main area.
 13. Themethod as claimed in claim 12, wherein the main vertical lines aredetermined using coordinate values of at least one right point in a lefteye area of the initial face area and coordinate values of at least oneleft point in a right eye area of the initial face area.
 14. The methodas claimed in claim 12, wherein the main horizontal lines are determinedusing coordinate values of at least one lower point in each of the eyeareas of the initial face area and coordinate values of at least oneupper point in the mouth area of the initial face area.
 15. The methodas claimed in claim 11, wherein the tilt parameter adjustment isachieved by controlling the lens in a predetermined direction until acentral y coordinate is positioned between two main horizontal lines ofthe initial face area.
 16. The method as claimed in claim 11, wherein atleast one of the pan parameter adjustment, the tilt parameteradjustment, and the zoom parameter adjustment is based on a result of aface element calculation performed by extracting a horizontal edge fromthe initial face area extracted from the input image, extracting eyeareas and a mouth area, and then calculating a main area based on theeye areas and the mouth area, the pan parameter adjustment is performedby controlling the second motor such that a center value of x coordinatevalues in the initial face area is positioned between coordinate valuesof at least one right point in a left eye area and coordinate values ofat least one left point in a right eye area, and the tilt parameteradjustment is performed by controlling the second motor such that acenter value of y coordinate values in the initial face area is betweencoordinate values of at least one lower point in the eye areas andcoordinate values of at least one upper point in the mouth area.
 17. Themethod as claimed in claim 11, wherein the main area is determined as aminimum outline square including the eye areas and the mouth area. 18.The method as claimed in claim 17, wherein the main area is defined by afirst point and a second point, an x coordinate value of the first pointis a smaller value selected from between an x coordinate value ola lefteye area and an x coordinate value of a left part of the mouth area, a ycoordinate value of the first point is a smaller value selected frombetween a y coordinate value of the left eye area and a y coordinatevalue of a right eye area, an x coordinate value of the second point isa larger value selected from between an x coordinate value of the righteye area and an x coordinate value of a right part of the mouth area,and a y coordinate value of the second point has a y coordinate value ofthe mouth area.